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Problem Description 


Earth Science studies such as: 


Climate change over multiple time scales 

Predicting crop production 

Monitoring land resources 

Understanding the impact of human activity on major Earth ecosystems 


¢ Addressed by using global and repetitive measurements provided 
by a wide variety of satellite remote sensing systems 


O 


Multiple-time or simultaneous observations of the same Earth features by 
different sensors 


Global measurements with remote sensing systems 


Complemented by regional and local measurements using ground and airborne 
sensors 


Addressed by using global and repetitive measurements provided by a 
wide variety of satellite remote sensing systems 


¢ Need to correlate and integrate all these complementary data 
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Example of International Measurements 


GPM Constellation of Satellites 


NPP (NASA/IPO) GPM Core Observatory 
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Image Registration Challenges 


¢ Remote Sensing vs. Medical or Other Imagery 
O Variety in the types of sensor data and the conditions of data acquisition 
O Size of the data 
o Lack of a known image model 


o Lack of well-distributed “fiducial points” resulting in lack of algorithms validation 


¢ Navigation Error 


¢ Atmospheric and Cloud Interactions 


~~ 


Three Landsat images over Virginia acquired in August, October, and November 1999 
(Courtesy: Jeffrey Masek, NASA Goddard Space Flight Center) 


Atmospheric and Cloud Interactions 


Atmospheric and 


Cloud Interactions 
Baja Peninsula, 
California; 4 different %- E 


times of the day (GOES-8) act 


(Reproduced from Le Moigne & = 
Eastman, 2005) 


Image Registration Challenges 
Multi- Temporal 


Mississippi and Ohio Rivers before & after Flood of Spring 2002 (Terra/MODIS) 


What is Image Registration? 


Image Registration/Feature-Based Precision Correction vs. Navigation or 
Model-Based Systematic Correction 

i, Orbital, Attitude, Platform/Sensor Geometric Relationship, Sensor Characteristics, Earth Model, etc. 
2, Navigation within a Few Pixels Accuracy 

3, Image Registration Using Selected Features (or Control Points) to Refine Geo-Location Accuracy 


Mathematical Framework 
fo) I1(x,y) and I2(x,y): images or image/map 
vY find the mapping (f,g) which transforms I] into I2: I2(x,y) = g(11(fx(x,y),fy(x,y)) 
a. f: spatial mapping 
b.  g: radiometric mapping 
O Spatial Transformations “f” 
Y Translation, Rigid, Affine, Projective, Perspective, Polynomial, ... 
o Radiometric Transformations “g” (Resampling) 
v Nearest Neighbor, Bilinear, Cubic Convolution, ... 


Algorithmic Framework (Brown, 1992) 
Search Space of potential transformations 


Feature Space of information extracted from the 2 datasets 
Similarity Metric used to match the 2 sets of features 
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Search Strategy to find the optimal transformation 
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Algorithmic Framework (Brown, 1992) 

Search Space of potential transformations 

Feature Space of information extracted from the 2 datasets 
Similarity Metric used to match the 2 sets of features 
Search Strategy to find the optimal transformation 
Resampling Method to create the corrected image 
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Validation Method to evaluate the accuracy of the registration 


Operational Solutions 
The Landsat ETM+ Example 


Sensor Knowledge 
o Sensor geometry 

o Sensor to platform 
O Orbit 

o Terrain data (DEM) 
o Radiometric model 
Geodetic accuracy 


o Database of GCPs derived from USGS 


ata 
o Normalized correlation 
o Updates navigation models 
o Results: RMSE ~54m 


Band-to-band registration 


o Selected tie-points in high-freq. arid ~~’ 


regions 
O Normalized correlation 


Sees ixel by second order fit to 3x3 
neighborhood 


o —_ Result: 0.1 to 0.2 subpixel 
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Operational Solutions 
Normalized Cross-Correlation (NCC) Often Used 


ASTER Terra | 15m-90m NCC w/ DEM 


GOES GOES I-M 1km-8km NCC w/ vector | Bi-section 
coastlines search 


MODIS 250m-1km |NCCw/DEM _ | Fixed grid 
SPOT NCC w/DEM __| Not described 
ETM+ Landsat-7 15m-60m NCC to arid Fit to surface 
region CPs 
VEGETATION | SPOT NCC w/ DEM _ | Not described 


Image Registration Algorithm 
Classifications 


Area-Based vs. Feature-Based 


o Often Combination of Area- and Feature-Based 


Alternate Classification: 

o Manual Registration 

o Correlation-Based Methods 

o Fourier-Domain and Other Transform-Based Methods 

o Mutual Information and Distribution-Based Approaches 
o Feature-Point Methods 


o Contour- and Region-Based Approaches 


Feature Extraction 


Features: 
o Gray levels 
o Salient points - Matched point-to-point or globally 
" Edge or edge-like, e.g., Sobel, Canny 
= Fourier coefficients 
=" Gabor, Wavelets, Directional Gabor or Wavelets, Shearlets, etc. 
=" Corners, e.g., Kearny, Harris and Stephens, Shi and Tomasi 


o Lines (Hough and Generalized), Contours (Govindu et al), Regions 
(Region Segmentation, e.g., Tilton) 


= Marked Point Processes (MPP): probabilistic framework with configuration space 
consisting of an unknown number of parametric objects 


o Scale invariant feature transform (SIFT-Lowe) and variants, e.g., 
Speeded Up Robust Features (SURF) 


o More recently, Neural Networks (NN) have been used for registration 


Similarity Metrics 


e Cross-correlation 


O Maximize cross-correlation over image overlap 
M-1N-1 


L(x yo (ny)= > YL (mn)L(x+m,y +n) 


m=0 n=0 
¢ Normalized cross-correlation (NCC) 
O Maximize normalized cross-correlation 
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¢ Mutual information (MI): 
Maximizes the degree of statistical dependence between the images 


MI(J,,1,) )= DP ( £12) ol Pht) (1-82) 


Py, (g,)- Py, (g,) 
or using histograms, maximizes 
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where / is the sum of all oa entries, i.e., number of 
pixels (in overlapping subimage) 


Similarity Metrics (cont.) 
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MI vs. £,-norm and NCC applied to 
Landsat-5 images 


Other Similarity Metrics 


Partial Hausdorff distance (PHD): 
Hy (1) min, -,, dist(p,, p,), 


yeh 


where 1< K <|J,|(Huttenlocher er al, Mount et al) 


_ dist(a, 1,)° 


where w,(a) denotes the weight of point a, and w,(a) = el ae 
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Image Matching Strategies 


Matching strategies matched with feature extraction 
techniques 


Some methods: 


O 


O 


O 


O 


O 


O 


Exhaustive Search 

FFT/Phase Correlation — Fourier Mellin Transform 
Optimization: 

» Steepest Gradient Descent 

= Levenberg-Marquart 

# Stocchastic Gradient 

Robust Feature Matching (RFM) 

Genetic algorithms (including binary shapes) 

Neural Networks (esp. for quantum & cognitive computing) 


Global or local registration 


Various image representations, e.g., Multi-resolution and 
quadtrees 


some Recent Image Registration Results 
Wavelet and Wavelet-Like Based Algorithms 


Edge, Wavelet and Wavelet-Like Based Registration Framework 


Features ¢ Wavelets are fundamentally isotropic, 1.e., 
no directional sensitivity 


mbar ) arnt ist * Generalization of wavelets to be 
easure 3 . 

anisotropic => Shearlets, which refine the 
wavelet construction by including a 


directional component 
Strategy 


Wavelet Features 
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SAR image (1024x1024) _Shearlet Features 


Landsat Warped and Noise Experiments 


Geometrically warped synthetic 
input images. The full source image 
is 1024 x 1024 Landsat-5 TM image 
from the Mount Hood are. The 
extracted images are 256 x 256. 


256 x 256 Landsat-7 ETM+ images of 
Washington, DC, (left) without and 
(right) with Gaussian noise added. The 
parameters for the noise are mean p = 
0 and variance o2 = 0.05 


Shearlet-Based Registration Results 
As a Function of Warp 


Comparison of Registration Algorithms for Landsat-TM Geometrically Warped Synthetic Experiments 
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Shearlet-Based Registration Results 
As a Function of Noise 


Comparison of Registration Algorithms for Noisy Landsat-ETM+ Synthetic Experiments (Variance = 0.05) 
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Multimodal Experiments 


1024 x 1024 images of (left) ETM+ 
Infared/Red band and (right) 
Near-Infared/NIR band of the Konza 


Prairie 


Pixels computed by SIFT in the 
LIDAR shaded-relief (left) and 3 
optical (right) images of Washington (age 
State, connected by line segments. 
Note the lack of correspondence; 
such points are unsuitable for a 
registration algorithm. 
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Shearlet-Based Registration Results 


For LIDAR Data 


Comparison of Registration Algorithms for LIDAR Warped Synthetic Experiments 
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Shearlet-Based Registration Results 
Multimodal Experiments 


Comparison of Registration Algorithms for ETM+ Infrared to NIR Multimodal Experiments 
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Shearlet-Based Registration Results 
Multimodal Experiments (cont.) 


Comparison of Registration Algorithms for LIDAR to Optical Multimodal Experiments 
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Preliminary Image Registration Results 
Using Artificial Neural Networks 


Feed-forward neural network (FF-NN) for * Using Discrete Cosine Transform (DCT) 
subpixel accuracy 


coefficients as input provides subpixel 

Output Layer: Tx, Ty, @, Scale accuracy 

¢ Input: 100 DCT coefficients from 
reference image + 100 DCT coefficients 
from test image 

¢ Output: Transformation Variables (Tx, Ty, 
0,s ) 

¢ Score: Subpixel registration accuracy if 

Continuous DCT Values (non-binary) mean RMS error < 1.0 per pixel 


Subpixel accuracy on 50% of the test images in < 500 training epochs. Running for longer 
increases accuracy 
Training set must be large enough to capture the range of values for rotation/translation in the test 
set 
¢ Training set of 100 images randomly rotated/translated from a source image is enough to learn: 
+/- 45° rotation coupled with +/- 10 pixels translation 
* Training set of 300 images is enough to learn: 
+/- 1202 rotation, no translation 
+/- 80 pixels translation, no rotation 


Current experiments using Deep Belief Networks and Restricted Boltzman Machines 


Conclusions 


Brief introduction to remote sensing image registration and its 
main components: 

o Feature Extraction 

o Similarity Metrics 


o Search Strategies 


Components combined appropriately and adapted to: 
o Type of data (e.g., edge- vs. texture-rich) 

o Size of data and computational resource needed 

o Required accuracy 

o _ Initial conditions 

Future Work: 

o Systematic assessment of various algorithms 


o Creating benchmark datasets 


